The Evidence Framework applied to Classi cation Networks

Three Bayesian ideas are presented for supervised adaptive classiers. First, it is argued that the output of a classier should be obtained by marginalising over the posterior distribution of the parameters; a simple approximation to this integral is proposed and demonstrated. This involves a `moderation' of the most probable classi-er's outputs, and yields improved performance. Second, it is demonstrated that the Bayesian framework for model comparison described for regression models in (MacKay, 1992a, 1992b) can also be applied to classication problems. This framework successfully chooses the magnitude of weight decay terms, and ranks solutions found using dierent numbers of hidden units. Third, an information{based data selection criterion is derived and demonstrated within this framework. 1 Introduction A quantitative Bayesian framework has been described for learning of mappings in feedfor-ward networks (MacKay, 1992a, 1992b). It was demonstrated that thisèvidence' framework could successfully choose the magnitude and type of weight decay terms, and could choose between solutions using dierent numbers of hidden units. The framework also gives quan-tied error bars expressing the uncertainty in the network's outputs and its parameters. In (MacKay, 1992c) information{based objective functions for active learning were discussed within the same framework. These three papers concentrated on interpolation (regression) problems. Neural networks can also be trained to perform classication tasks.